Skip to content

changelog: source bundle entries from the CDN with a per-product registry#3470

Merged
cotti merged 2 commits into
mainfrom
changelog_bundle_s3_source
Jun 19, 2026
Merged

changelog: source bundle entries from the CDN with a per-product registry#3470
cotti merged 2 commits into
mainfrom
changelog_bundle_s3_source

Conversation

@cotti

@cotti cotti commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Note

Stacked on changelog_directive_s3. Base this against that branch and merge it first; the diff shown here is only this change.

Summary

Source the individual changelog entries that make up a bundle from the public CDN by default (scoped to the bundle's product), instead of the local docs/changelog/ folder. This makes bundles reflect what was actually published to S3 — i.e. with private references scrubbed — and decouples bundle creation from a checkout that happens to hold every entry.

The bundle filter is content-based: whether an entry belongs in a bundle depends on the products / prs / issues fields inside the YAML, not on the file name. CloudFront has no ListObjects, so we publish a small per-product index on upload and the client enumerates → fetches → filters.

What changed

  • Per-product entry registry: upload now writes {product}/changelog/registry.json alongside the existing bundle registry (RegistryBuilder / RegistryKey); the scrubber key allow-list recognizes it.
  • CDN entry sourcing: new CdnChangelogEntryFetcher downloads the entry index and each entry; CDN base resolution is centralized in ChangelogCdn (shared with the changelog directive's cdn: mode).
  • Opt-out + safe fallback: bundle.use_local_changelogs: true forces the local folder, and we fall back to local automatically when no concrete product can scope the per-product CDN fetch.
  • --plan cdn_url: changelog bundle --plan now emits cdn_url ({base}/{product}/bundle/{file}) so CI can poll for the scrubbed bundle. docs/cli-schema.json regenerated.
  • No silent gaps: a registry-listed entry that hasn't propagated to the CDN yet is retried with short backoff + cache-busting (defeats a CloudFront-cached 404); a persistent miss fails the bundle rather than shipping an incomplete release.

Verification

  • dotnet format --verify-no-changes: clean
  • dotnet publish -c Release (docs-builder): 0 trim/AOT warnings
  • Elastic.Documentation.Configuration.Tests (409/409) and Elastic.Changelog.Tests (732/732): pass
  • cli-schema.json matches -- __schema output

Test plan

  • upload publishes {product}/changelog/registry.json and individual entries
  • bundle (default) fetches scrubbed entries from the CDN and applies the content filter
  • bundle with use_local_changelogs: true uses the local folder
  • bundle --plan surfaces a correct cdn_url
  • A not-yet-propagated entry recovers via retry; a persistently missing entry fails the run

Made with Cursor

@cotti cotti requested review from a team as code owners June 5, 2026 00:29
@cotti cotti requested a review from Mpdreamz June 5, 2026 00:29
@cotti cotti temporarily deployed to integration-tests June 5, 2026 00:29 — with GitHub Actions Inactive
@cotti cotti added the fix label Jun 5, 2026
@cotti cotti self-assigned this Jun 5, 2026

@Mpdreamz Mpdreamz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few code-quality notes, then a broader design question worth discussing before this merges.

Code issues

HttpClient is never disposed in CdnChangelogEntryFetcher.
The class allocates an HttpClient in a field initializer but doesn't implement IDisposable, so the connection pool leaks in any reuse scenario (tests, future service mode). Should implement IDisposable and dispose the client, or accept an IHttpClientFactory.

Fetch() blocks the calling thread for all HTTP I/O.
FetchRegistry and FetchText use the synchronous HttpClient.Send() overload. Everything else in the service layer is async; this blocks a thread-pool thread for the full CDN round-trip per product.

Multi-paragraph XML doc comments on private methods.
FetchCdnEntries, ResolveCdnBundleUrl, and ResolvePrimaryProduct all have verbose <summary> blocks. Project style is one short line max (or none) on private methods.

PlanBundleAsync duplicates the CDN-producibility check.
The plan path and the bundle path compute "is a product resolvable?" with different expressions at different pipeline stages (pre- vs post-ApplyConfigDefaults). Currently correct, but there is no comment explaining the difference, so they are likely to diverge as the code evolves.


Design question — should this be declarative in docset.yml?

The current approach fetches changelog entries imperatively inside changelog bundle, driven by a use_local_changelogs escape hatch in changelog.yml. That works, but it means:

  • The fetch happens deep in the bundle command where concurrency is ad-hoc (synchronous loop over entries per product).
  • The build cannot validate or prefetch CDN requirements up-front — it discovers the dependency at execution time.
  • The changelog directive has nowhere to point the user on a fetch error; it can only say "fetch failed" rather than "declare this in your config".

Compare how crosslinks work: repos are declared in docset.yml under cross_links, docs-builder knows about the external dependency at startup, can fail fast if the index is unreachable, and can fetch all link indexes concurrently before any page is rendered.

A similar pattern for changelog sourcing might look like:

# docset.yml
release_notes:
  - repository: elastic/elasticsearch
  - repository: elastic/kibana

With that declaration:

  • docs-builder knows at startup which CDN products to pull, can fetch all registries and entries concurrently, and can fail fast before any directive tries to render.
  • The changelog directive just references an already-loaded in-memory set — no per-directive HTTP.
  • If a directive references a product not declared in docset.yml, it emits a clear error pointing the user at the config key rather than a raw CDN failure at render time.
  • changelog bundle reads the same declaration to know which products to source from the CDN, keeping the config in one place.

The use_local_changelogs flag would still make sense as an escape hatch, but the primary sourcing decision would be declared rather than implied by the bundle command's product filter.

Is this the direction you want to go, or is there a reason to keep it embedded in changelog.yml / the bundle command?

@cotti

cotti commented Jun 19, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto the updated #3436 and reconciled with the opt-in model; addressed the HttpClient feedback

Rebase

HttpClient (review feedback)

  • CdnChangelogEntryFetcher is now IDisposable and fully async: a shared HttpClient in production, an owned client disposed with the instance only when a test injects a handler, and async retry backoff (Task.Delay) instead of blocking sleeps. Propagated through FetchCdnEntriesAsync.

Declared-gate sourcing

  • changelog bundle sources entries from the CDN only when every in-scope product is declared under release_notes in docset.yml — the same opt-in declaration the directive consumes. Otherwise (or with bundle.use_local_changelogs, --directory, or no concrete product in scope) it falls back to local folder sourcing.
  • The same gate drives the --plan needs_network output, so the planning step and the Docker bundle run stay in agreement.
  • Unified product resolution across the run + plan paths; trimmed verbose private-method XML docs.

Docs / tests — added a bundle.use_local_changelogs entry + an "Entry sourcing" section to configure-changelogs-ref.md, a producer-side declared-gate section to the registry dev doc, and declared-gate test coverage (OptionMode_UndeclaredProduct_FallsBackToLocal, Plan_ProfileMode_UndeclaredProduct_ReturnsNoNetwork).

Client / docs-actions hand-off

  • Opt-in is now required per product: elastic-otel-java declares edot-java; cloud declares its rendered products (e.g. cloud-hosted, cloud-serverless, cloud-enterprise).
  • docs-actions needs no workflow change — the registry refresh is already a side-effect of changelog upload, and --plan reflects the gate. Just publish each product's registry before declaring it (strict fail-fast).

@cotti cotti force-pushed the changelog_bundle_s3_source branch from 09bca38 to 7e3087d Compare June 19, 2026 03:38
@cotti cotti temporarily deployed to integration-tests June 19, 2026 03:38 — with GitHub Actions Inactive
@cotti cotti force-pushed the changelog_bundle_s3_source branch from 7e3087d to 26fc895 Compare June 19, 2026 13:08
@cotti cotti temporarily deployed to integration-tests June 19, 2026 13:08 — with GitHub Actions Inactive
Base automatically changed from changelog_directive_s3 to main June 19, 2026 13:21
cotti and others added 2 commits June 19, 2026 10:22
…stry

Source the individual changelog entries that make up a bundle from the
public CDN by default, scoped to the bundle's product(s), instead of the
local folder. Because the bundle filter is content-based (an entry's
products/prs/issues live inside the YAML, not its name) and CloudFront has
no ListObjects, a per-product entry index ({product}/changelog/registry.json)
is published on upload so the client can enumerate then fetch+filter.

- Add bundle.use_local_changelogs opt-out, plus automatic local fallback
  when no concrete product can scope the per-product CDN fetch.
- Extend RegistryBuilder/RegistryKey to write and pass-through the entry
  index (scrubber recognizes {product}/changelog/registry.json).
- Add CdnChangelogEntryFetcher and centralize CDN base resolution in
  ChangelogCdn (shared with the changelog directive's cdn: mode).
- Emit cdn_url from `changelog bundle --plan` so CI can poll for the
  scrubbed bundle ({base}/{product}/bundle/{file}).
- Harden entry sourcing: a registry-listed entry that has not yet
  propagated to the CDN is retried with short backoff and cache-busting;
  a persistent miss fails the bundle instead of silently shipping an
  incomplete release.

dotnet format, AOT publish (0 trim/AOT warnings), and the affected unit
tests all pass; cli-schema.json regenerated for the new --plan output.

Co-authored-by: Cursor <cursoragent@cursor.com>
Make `changelog bundle` source entries from the CDN only when every
in-scope product is declared under `release_notes` in docset.yml, matching
the directive's opt-in model. Undeclared products — and `use_local_changelogs`
or `--directory` — fall back to local sourcing. The same declared-gate drives
the `--plan` `needs_network` output so the planning step and bundle run agree.

Also make CdnChangelogEntryFetcher IDisposable and fully async (shared
HttpClient in production, owned client only for injected test handlers, async
retry backoff), addressing the HttpClient review feedback.

Updates bundle configuration/registry docs and tests.

Co-authored-by: Cursor <cursoragent@cursor.com>
@cotti cotti force-pushed the changelog_bundle_s3_source branch from 26fc895 to 332a2af Compare June 19, 2026 13:23
@cotti cotti temporarily deployed to integration-tests June 19, 2026 13:23 — with GitHub Actions Inactive
@cotti cotti enabled auto-merge (squash) June 19, 2026 13:24
@coderabbitai

coderabbitai Bot commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Caution

Review failed

An error occurred during the review process. Please try again later.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch changelog_bundle_s3_source
  • 🛠️ Update Documentation

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@cotti cotti merged commit 654fcce into main Jun 19, 2026
24 of 25 checks passed
@cotti cotti deleted the changelog_bundle_s3_source branch June 19, 2026 13:34
@coderabbitai

coderabbitai Bot commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Caution

Review failed

Pull request was closed or merged during review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Enterprise

Run ID: 5f0f5686-d498-4073-aaa2-601631546034

📥 Commits

Reviewing files that changed from the base of the PR and between 86a8ec4 and 332a2af.

📒 Files selected for processing (22)
  • docs/cli-schema.json
  • docs/contribute/configure-changelogs-ref.md
  • docs/development/changelog-bundle-registry.md
  • src/Elastic.Documentation.Configuration/Changelog/BundleConfiguration.cs
  • src/Elastic.Documentation.Configuration/Changelog/ChangelogConfigurationLoader.cs
  • src/Elastic.Documentation.Configuration/Changelog/ChangelogConfigurationYaml.cs
  • src/Elastic.Documentation.Configuration/ReleaseNotes/CdnChangelogEntryFetcher.cs
  • src/services/Elastic.Changelog/Bundling/ChangelogBundlingService.cs
  • src/services/Elastic.Changelog/Bundling/ChangelogEntryMatcher.cs
  • src/services/Elastic.Changelog/Uploading/ChangelogUploadService.cs
  • src/services/Elastic.Changelog/Uploading/RegistryBuilder.cs
  • src/services/Elastic.Changelog/Uploading/RegistryKey.cs
  • src/tooling/docs-builder/Commands/ChangelogCommand.cs
  • tests/Elastic.Changelog.Tests/Changelogs/BundleCdnSourcingTests.cs
  • tests/Elastic.Changelog.Tests/Changelogs/BundleChangelogsTests.cs
  • tests/Elastic.Changelog.Tests/Changelogs/BundlePlanTests.cs
  • tests/Elastic.Changelog.Tests/Changelogs/BundleProfileGitHubReleaseTests.cs
  • tests/Elastic.Changelog.Tests/Changelogs/ChangelogConfigurationTests.cs
  • tests/Elastic.Changelog.Tests/Changelogs/ChangelogTestBase.cs
  • tests/Elastic.Changelog.Tests/Uploading/RegistryBuilderTests.cs
  • tests/Elastic.Changelog.Tests/Uploading/RegistryKeyTests.cs
  • tests/Elastic.Documentation.Configuration.Tests/ReleaseNotes/CdnChangelogEntryFetcherTests.cs

📝 Walkthrough

Walkthrough

This PR adds CDN-sourced changelog bundling to the changelog bundle command. A new bundle.use_local_changelogs configuration setting (default false) and a CdnChangelogEntryFetcher class are introduced. The fetcher downloads per-product changelog/registry.json manifests and individual YAML entries from a CDN, with retry/backoff, cache-busting, and path-traversal filename guards. Sourcing is gated by a "declared-gate" check: all in-scope products must be listed under release_notes in docset.yml; otherwise the bundler falls back to local files. RegistryBuilder gains a RegistryScope enum to write both {product}/registry.json and {product}/changelog/registry.json manifests. BundlePlanResult gains a CdnUrl field emitted as a cdn_url GitHub Actions step output. Existing local-sourcing tests are updated with use_local_changelogs: true.

Suggested labels

feature

🚥 Pre-merge checks | ✅ 2
✅ Passed checks (2 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch changelog_bundle_s3_source
  • 🛠️ Update Documentation

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants